Methods and systems for providing visuospatial information

ABSTRACT

Methods and systems for providing feedback during a medical procedure. The 3D position and orientation of a tracked tool are determined, relative to a site of the medical procedure, based on tracking information received from a tracking system. The 3D position and orientation are mapped to a common coordinate space, to determine the 3D position and orientation relative to a field-of-view (FOV) of an optical camera that is capturing an optical image of the site. Navigational information associated with the 3D position and orientation is determined. A virtual representation of the navigational information overlaid on the FOV and displayed. The displayed virtual representation is updated when the 3D position and orientation of the tracked tool changes or when the FOV changes, in accordance with the change.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/650,253 filed Jul. 14, 2017 entitled METHODS AND SYSTEMS FORPROVIDING VISUOSPATIAL INFORMATION, the contents of which are herebyexpressly incorporated into the present application by reference intheir entirety.

FIELD

The present disclosure relates to methods and systems for providingintraoperative navigational feedback. In particular, the presentdisclosure relates to providing visuospatial navigational information,using a tracking system and visual overlay.

BACKGROUND

In an example neurosurgical procedure, a surgeon or a robotic surgicalsystem may perform a port-based minimally-invasive procedure involvingtumor resection in the brain. A goal of the procedure typically includesminimizing trauma to healthy tissue, such as the intact white and greymatter of the brain. Trauma may occur, for example, due to contact ofhealthy tissue with the access port, stress to the brain matter,unintentional impact with surgical devices, and/or accidental resectionof healthy tissue. In order to reduce trauma, the surgeon should haveaccurate information, including depth information, about where thesurgical tools are relative to the surgical site of interest.

Conventional systems may not provide information about the surgical sitein sufficient detail. For example, in conventional procedures, thesurgeon is typically provided with a view of the site of interest via acamera or eyepiece of a microscope, endoscope or exoscope. Thistypically provides only a real-life view of the actual site, without anyadditional visuospatial information that might help the surgeon.Instead, the surgeon is required to turn to other screens or monitorsfor additional information, or rely on their own trained visuospatialabilities. This can be taxing to the surgeon and may lead to longerprocedures and greater risk of accidental trauma to healthy tissue.

SUMMARY

In some examples, the present disclosure provides a system for providingfeedback during a medical procedure. The system includes a trackingsystem configured to obtain tracking information about three-dimensional(3D) position and orientation of a tracked tool during the medicalprocedure. The system also includes the tracked tool coupled to trackingmarkers to enable tracking of the tracked tool by the tracking system.The system also includes a camera for capturing an optical image of afield-of-view (FOV) of a site during the medical procedure. The systemalso includes a display for displaying the optical image of the FOV. Thesystem also includes a processor coupled to receive input data from thetracking system and the camera, and coupled to transmit output data fordisplay on the display. The processor is configured to determine the 3Dposition and orientation of the tracked tool, relative to the site,based on the tracking information. The processor is also configured tomap the 3D position and orientation to a common coordinate space, todetermine the 3D position and orientation relative to the FOV. Theprocessor is also configured to determine navigational informationassociated with the 3D position and orientation. The processor is alsoconfigured to cause the display to display a virtual representation ofthe navigational information overlaid on the FOV. The processor is alsoconfigured to update the displayed virtual representation by: when the3D position and orientation of the tracked tool changes, updating thedisplayed virtual representation in accordance with the changed 3Dposition and orientation; or when the FOV changes, updating thedisplayed virtual representation to follow the changed FOV.

In some examples, the present disclosure provides a method for providingfeedback during a medical procedure. The method includes determining the3D position and orientation of a tracked tool, relative to a site of themedical procedure, based on tracking information received from atracking system that is tracking the tracked tool. The method alsoincludes mapping the 3D position and orientation to a common coordinatespace, to determine the 3D position and orientation relative to afield-of-view (FOV) of a camera that is capturing an optical image ofthe site. The method also includes determining navigational informationassociated with the 3D position and orientation. The method alsoincludes causing a display to display a virtual representation of thenavigational information overlaid on the FOV. The method also includesupdating the displayed virtual representation by: when the 3D positionand orientation of the tracked tool changes, updating the displayedvirtual representation in accordance with the changed 3D position andorientation; or when the FOV changes, updating the displayed virtualrepresentation to follow the changed FOV.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made, by way of example, to the accompanyingdrawings which show example embodiments of the present application, andin which:

FIG. 1 illustrates the insertion of an access port into a human brain,for providing access to internal brain tissue during an example medicalprocedure;

FIG. 2 shows an example navigation system to support image guidedsurgery;

FIG. 3 is a diagram illustrating system components of an examplenavigation system;

FIG. 4 is a block diagram illustrating an example control and processingsystem that may be used in the navigation system of FIG. 2;

FIG. 5 is a diagram illustrating co-registration of two coordinatespaces;

FIG. 6A is a flowchart illustrating an example method for providingintraoperative visuospatial information;

FIG. 6B shows an example display of a captured image including a cursorfor interacting with the image;

FIG. 7A shows an example display of a captured image including visualrepresentation of selected 3D points;

FIG. 7B shows example displays illustrating persistence of a visualrepresentation of navigational information when the zoom level of theimage changes;

FIG. 7C illustrates how depth information may be calculated for aselected 3D point;

FIG. 8 shows an example display of a captured image including visualrepresentation of a selected boundary of interest;

FIG. 9A shows an example display of a captured image including visualrepresentation of a selected 3D point and 3D orientation;

FIGS. 9B-9G illustrate an example of how selected 3D points andorientations are provided as visuospatial information;

FIG. 10 shows an example display of a captured image includingnavigational information within a selected region of interest;

FIG. 11 shows an example display of imaging data including an overlay ofreal-time captured images in a selected region of interest;

FIG. 12 shows an example display of a captured image including visualmodification within a selected region of interest;

FIG. 13 shows an example display of a captured image including visualrepresentation of selected reference lines;

FIG. 14 shows an example display of a captured image including a visualrepresentation of a reference orientation;

FIG. 15 shows an example display of a captured image including visualrepresentation of planned targets;

FIG. 16 shows an example display of a captured image including anoverlay of a user interface; and

FIG. 17 shows example displays of different image modalities,illustrating persistence of visual representation of navigationalinformation across different image modalities.

Similar reference numerals may have been used in different figures todenote similar components.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The systems and methods described herein may be useful in medicalprocedures, including surgical procedures. The present disclosureprovides examples in the field of neurosurgery, such as for oncologicalcare, treatment of neurodegenerative disease, stroke, and brain trauma.Persons of skill will appreciate the ability to extend these concepts toother conditions or fields of medicine. For example, the presentdisclosure may also be applicable to the field of spinal surgery ororthopedic surgery, among others. It should be noted that while thepresent disclosure describes examples in the context of neurosurgery,the present disclosure may be applicable to other procedures that maybenefit from providing visuospatial information during the medicalprocedure.

Visuospatial information that may be provided by methods and systemsdisclosed herein include navigational information, for example includingdimensional information and trajectory information. Dimensionalinformation may include, for example, information about the position andorientation of a tracked tool or target, diameter of a tumour, depth ofa cavity, size of a pedicle, angle of approach and/or depth of a target.Trajectory information may include information related to a plannedtrajectory including, for example, visual indication of the plannedtrajectory, planned targets and/or updates to the planned trajectory asthe view of the site changes.

Further, a surgeon (or other operator) may be able to modify thevisuospatial information, for example to mark a point, region orboundary of interest, to change the visual presentation (e.g., contrast,sharpness and/or color) and/or restrict image processing or visuospatialinformation to a selected area, point, shape and/or property (e.g., toreduce computation time and/or reduce mental load).

Various example apparatuses or processes will be described below. Noexample embodiment described below limits any claimed embodiment and anyclaimed embodiments may cover processes or apparatuses that differ fromthose examples described below. The claimed embodiments are not limitedto apparatuses or processes having all of the features of any oneapparatus or process described below or to features common to multipleor all of the apparatuses or processes described below. It is possiblethat an apparatus or process described below is not an embodiment of anyclaimed embodiment.

Furthermore, numerous specific details are set forth in order to providea thorough understanding of the disclosure. However, it will beunderstood by those of ordinary skill in the art that the embodimentsdescribed herein may be practiced without these specific details. Inother instances, well-known methods, procedures and components have notbeen described in detail so as not to obscure the embodiments describedherein.

As used herein, the terms, “comprises” and “comprising” are to beconstrued as being inclusive and open ended, and not exclusive.Specifically, when used in the specification and claims, the terms,“comprises” and “comprising” and variations thereof mean the specifiedfeatures, steps or components are included. These terms are not to beinterpreted to exclude the presence of other features, steps orcomponents.

As used herein, the term “exemplary” or “example” means “serving as anexample, instance, or illustration,” and should not be construed aspreferred or advantageous over other configurations disclosed herein.

As used herein, the terms “about”, “approximately”, and “substantially”are meant to cover variations that may exist in the upper and lowerlimits of the ranges of values, such as variations in properties,parameters, and dimensions. In one non-limiting example, the terms“about”, “approximately”, and “substantially” mean plus or minus 10percent or less.

Unless defined otherwise, all technical and scientific terms used hereinare intended to have the same meaning as commonly understood by one ofordinary skill in the art. Unless otherwise indicated, such as throughcontext, as used herein, the following terms are intended to have thefollowing meanings:

As used herein, the phrase “access port” refers to a cannula, conduit,sheath, port, tube, or other structure that is insertable into asubject, in order to provide access to internal tissue, organs, or otherbiological substances. In some embodiments, an access port may directlyexpose internal tissue, for example, via an opening or aperture at adistal end thereof, and/or via an opening or aperture at an intermediatelocation along a length thereof. In other embodiments, an access portmay provide indirect access, via one or more surfaces that aretransparent, or partially transparent, to one or more forms of energy orradiation, such as, but not limited to, electromagnetic waves andacoustic waves.

As used herein the phrase “intraoperative” refers to an action, process,method, event or step that occurs or is carried out during at least aportion of a medical procedure. Intraoperative, as defined herein, isnot limited to surgical procedures, and may refer to other types ofmedical procedures, such as diagnostic and therapeutic procedures.

As used herein the phrase “preoperative” refers to an action, process,method, event or step that occurs prior to the start of a medicalprocedure. Preoperative, as defined herein, is not limited to surgicalprocedures, and may refer to other types of medical procedures, such asdiagnostic and therapeutic procedures. Planning a medical procedure maybe considered to be preoperative.

Some embodiments of the present disclosure include imaging devices thatare insertable into a subject or patient for imaging internal tissues,and methods of use thereof. Some embodiments of the present disclosurerelate to minimally invasive medical procedures that are performed viaan access port or retractor tube, whereby surgery, diagnostic imaging,therapy, or other medical procedures (e.g., minimally invasive medicalprocedures) are performed based on access to internal tissue through theaccess port or retractor tube.

FIG. 1 illustrates the insertion of an access port into a human brain,for providing access to internal brain tissue during a medicalprocedure. In FIG. 1, an access port 102 is inserted into a human brain104, providing access to internal brain tissue. The access port 102 mayinclude such instruments as catheters, surgical probes, or cylindricalports such as the NICO BrainPath™. Surgical tools and instruments maythen be inserted within the lumen of the access port in order to performsurgical, diagnostic or therapeutic procedures, such as resectingtumors, as necessary.

The present disclosure applies equally well to catheters, deep brainstimulation (DBS) needles, a biopsy procedure, and also to biopsiesand/or catheters in other medical procedures performed on other parts ofthe body, as well as to medical procedures that do not use an accessport, including non-neural medical procedures, such as spinalprocedures.

In the example of a port-based surgery, a straight or linear access port102 is typically guided down a sulcal path of the brain. Surgicalinstruments would then be inserted down the access port 102. Opticaltracking systems, used in the medical procedure, track the position of apart of the instrument that is within line-of-sight of the opticaltracking camera using by the tracking system. Other tracking systems maybe used, such as electromagnetic, optical, or mechanical based trackingsystems, including tracking systems that use multiple tracking camerasor do not use any tracking camera.

In FIG. 2, an exemplary navigation system environment 200 is shown,which may be used to support navigated image-guided surgery. As shown inFIG. 2, a surgeon 201 conducts a surgery on a patient 202 in anoperating room (OR) environment. A medical navigation system 205 mayinclude an equipment tower, tracking system, displays and trackedinstruments to assist the surgeon 201 during the procedure. An operator203 may also be present to operate, control and provide assistance forthe medical navigation system 205.

FIG. 3 shows a diagram illustrating components of the example medicalnavigation system 205. The disclosed methods and systems for providingvisuospatial information may be implemented in the context of themedical navigation system 205. The medical navigation system 205 mayinclude one or more displays 311 for displaying still and/or videoimages (e.g., a live video image of the surgical field and/or 2D or 3Dimages obtained preoperatively), an equipment tower 301, and apositioning system 302 (e.g., a mechanical arm), which may support anoptical scope 304 (which may also be referred to as an external scope).One or more of the displays 311 may include a touch-sensitive displayfor receiving touch input. The equipment tower 301 may be mounted on aframe (e.g., a rack or cart) and may contain a power supply and acomputer or controller that may execute planning software, navigationsoftware and/or other software to manage the positioning system 302 andtracked instruments. In some examples, the equipment tower 301 may be asingle tower configuration operating with multiple displays 311, howeverother configurations may also exist (e.g., multiple towers, singledisplay, etc.). Furthermore, the equipment tower 301 may also beconfigured with a universal power supply (UPS) to provide for emergencypower, in addition to a regular AC adapter power supply.

A portion of the patient's anatomy may be held in place by a holder. Forexample, in the context of a neurosurgical procedure, the patient's headand brain may be held in place by a head holder 317. An access port 102and associated introducer 310 may be inserted into the head, to provideaccess to a surgical site in the head. The optical scope 304 may beattached to the positioning system 302, and may be used to view down theaccess port 102 at a sufficient magnification to allow for enhancedvisibility down the access port 102. The output of the optical scope 304may be received by one or more computers or controllers to generate aview that may be depicted on a visual display (e.g., one or moredisplays 311).

In some examples, the navigation system 205 may include a tracked tool320, which may include or be coupled to one or more markers 312 (alsoreferred to as tracking markers or fiducial markers) to enable trackingby a tracking camera of a tracking system 313 that is part of thenavigation system 205. As mentioned above, in various examples thetracking system 313 may have one tracking camera, multiple trackingcameras or no tracking camera. The tracking system 313 may provide, to aprocessor of the navigation system 205, tracking information indicatingthe position and orientation of the tracked tool 320, as describedfurther below. An example of a tracked tool 320 may be a pointing tool,which may be used to identify points (e.g., fiducial points or pointsbordering a craniotomy opening, as discussed below) on a patient. Forexample, an operator, typically a nurse or the surgeon 201, may use thepointing tool to identify the location of points on the patient 202, inorder to register the location of selected points on the patient 202 inthe navigation system 205. A tracked tool 320 may also be a suctiontool. In addition to providing suction, the distal end of the suctiontool may be used for pointing, similarly to the distal end of a pointingtool. It should be noted that a guided robotic system may be used as aproxy for human interaction. Guidance to the robotic system may beprovided by any combination of input sources such as image analysis,tracking of objects in the operating room using markers placed onvarious objects of interest, or any other suitable robotic systemguidance techniques.

One or more markers 312 may also be coupled to the introducer 310 toenable tracking by the tracking system 313, and the tracking system 313may provide, to a processor of the navigation system 205, trackinginformation indicating the position and orientation of the introducer310. In some examples, the markers 312 may be alternatively oradditionally attached to the access port 102. Other tools (not shown)may be provided with markers 312 to enable tracking by the trackingsystem 313.

In some examples, the tracking camera used by the tracking system 313may be a 3D infrared optical tracking stereo camera similar to one madeby Northern Digital Imaging (NDI). In some examples, the tracking system313 may be an electromagnetic system (not shown). An electromagnetictracking system may include a field transmitter and the tracking markers312 may include receiver coils coupled to the tool(s) 320 to be tracked.The known profile of the electromagnetic field and the known position ofreceiver coil(s) relative to each other may be used to infer thelocation of the tracked tool(s) 320 using the induced signals and theirphases in each of the receiver coils. Operation and examples of thistechnology is further explained in Chapter 2 of “Image-GuidedInterventions Technology and Application,” Peters, T.; Cleary, K., 2008,ISBN: 978-0-387-72856-7, incorporated herein by reference.

Tracking information of the positioning system 302 and/or access port102 may be determined by the tracking system 313 by detection of themarkers 312 placed on or otherwise in fixed relation (e.g., in rigidconnection) to any of the positioning system 302, the access port 102,the introducer 310, the tracked tool 320 and/or other tools.

The marker(s) 312 may be active or passive markers. Active markers mayinclude infrared emitters for use with an optical tracking system, forexample. Passive markers may include reflective spheres for use with anoptical tracking system, or pick-up coils for use with anelectromagnetic tracking system, for example.

The markers 312 may all be the same type or may include a combination oftwo or more different types. Possible types of markers that could beused may include reflective markers, radiofrequency (RF) markers,electromagnetic (EM) markers, pulsed or un-pulsed light-emitting diode(LED) markers, glass markers, reflective adhesives, or reflective uniquestructures or patterns, among others. RF and EM markers may havespecific signatures for the specific tools they may be attached to.Reflective adhesives, structures and patterns, glass markers, and LEDmarkers may be detectable using optical detectors, while RF and EMmarkers may be detectable using antennas. Different marker types may beselected to suit different operating conditions. For example, using EMand RF markers may enable tracking of tools without requiring aline-of-sight from the tracking camera to the markers 312, and using anoptical tracking system 313 may avoid additional noise from electricalemission and detection systems.

In some examples, the markers 312 may include printed or 3D designs thatmay be used for detection by an auxiliary camera, such as a wide-fieldcamera (not shown) and/or the optical scope 304. Printed markers mayalso be used as a calibration pattern, for example to provide distanceinformation (e.g., 3D distance information) to an optical detector.Printed identification markers may include designs such as concentriccircles with different ring spacing and/or different types of bar codes,among other designs. In some examples, in addition to or in place ofusing markers 312, the contours of known objects (e.g., the side of theaccess port 102) could be captured by and identified using opticalimaging devices and the tracking system 313.

The markers 312 may be captured by the tracking camera (which may be astereo camera) to give identifiable points for tracking the tool(s) 320.A tracked tool 320 may be defined by a grouping of markers 312, whichmay define a rigid body to the tracking system 313. This may in turn beused to determine the position and/or orientation in 3D of a trackedtool 320 in a virtual space. The position and orientation of the trackedtool 320 in 3D may be tracked in six degrees of freedom (e.g., x, y, zcoordinates and pitch, yaw, roll rotations), in five degrees of freedom(e.g., x, y, z, coordinate and two degrees of free rotation), buttypically tracked in at least three degrees of freedom (e.g., trackingthe position of the tip of a tool in at least x, y, z coordinates). Intypical use with the navigation system 205, at least three markers 312are provided on a tracked tool 320 to define the tool 320 in virtualspace, however it may be advantageous for four or more markers 312 to beused.

Camera images capturing the markers 312 may be logged and tracked, by,for example, a closed circuit television (CCTV) camera. The markers 312may be selected to enable or assist in segmentation in the capturedimages. For example, infrared (IR)-reflecting markers and an IR lightsource from the direction of the tracking camera may be used. An exampleof such an apparatus may be tracking devices such as the Polaris® systemavailable from Northern Digital Inc. In some examples, the spatialposition of the tracked tool 320 and/or the actual and desired positionof the positioning system 302 may be determined by optical detectionusing the tracking camera. The optical detection may be done using anoptical camera, rendering the markers 312 optically visible.

Different tracked tools and/or tracked targets may be provided withrespective sets of markers 312 in different configurations.Differentiation of the different tools and/or targets and theircorresponding virtual volumes may be possible based on the specificationconfiguration and/or orientation of the different sets of markers 312relative to one another, enabling each such tool and/or target to have adistinct individual identity within the navigation system 205. Theindividual identifiers may provide information to the navigation system205, such as information relating to the size and/or shape of the tool320 within the system 205. The identifier may also provide additionalinformation such as the tool's central point or the tool's central axis,among other information. The markers 312 may be tracked relative to areference point or reference object in the operating room, such as oneor more reference points on the patient 202.

The display 311 may provide output of the computed data of thenavigation system 205. In some examples, the output provided by thedisplay 311 may include axial, sagittal and coronal views of patientanatomy as part of a multi-view output. In some examples, the one ormore displays 311 may include an output device, such as a wearabledisplay device, to provide an augmented reality (AR) display of the siteof interest.

A guide clamp 318 (or more generally a guide) for holding the accessport 102 may be provided. The guide clamp 318 may allow the access port102 to be held at a fixed position and orientation while freeing up thesurgeon's hands. An articulated arm 319 may be provided to hold theguide clamp 318. The articulated arm 319 may have up to six degrees offreedom to position the guide clamp 318. The articulated arm 319 may belockable to fix its position and orientation, once a desired position isachieved. The articulated arm 319 may be attached or attachable to apoint based on the patient head holder 317, or another suitable point(e.g., on another patient support, such as on the surgical bed), toensure that when locked in place, the guide clamp 318 does not moverelative to the patient's head.

In a surgical operating room (or theatre), setup of a navigation systemmay be relatively complicated; there may be many pieces of equipmentassociated with the medical procedure, as well as elements of thenavigation system 205. Further, setup time typically increases as moreequipment is added. The surgeon 201 may be required to process many setsof information from different equipment during the medical procedure.Information may be primarily of a visual nature, and the surgeon 201 mayeasily be overwhelmed by the amount of information to be processed. Toassist in addressing this, the navigation system 205 may include twoadditional wide-field cameras to enable information to be overlaid on areal-time view of the site of interest. One wide-field camera may bemounted on the optical scope 304, and a second wide-field camera may bemounted on the tracking camera. Video overlay information can then beadded to displayed images, such as images displayed on one or more ofthe displays 300. The overlaid information may provide visuospatialinformation, such as indicating the physical space where accuracy of the3D tracking system is greater, the available range of motion of thepositioning system 302 and/or the optical scope 304, and/or othernavigational information, as discussed further below.

Although described in the present disclosure in the context ofport-based neurosurgery (e.g., for removal of brain tumors and/or fortreatment of intracranial hemorrhages (ICH)), the navigation system 205may also be suitable for one or more of: brain biopsy,functional/deep-brain stimulation, catheter/shunt placement (in thebrain or elsewhere), open craniotomies, and/orendonasal/skull-based/ear-nose-throat (ENT) procedures, as well asprocedures other than neurosurgical procedures. The same navigationsystem 205 may be used for carrying out any or all of these procedures,with or without modification as appropriate.

For example, the same navigation system 205 may be used to carry out adiagnostic procedure, such as brain biopsy. A brain biopsy may involvethe insertion of a thin needle into a patient's brain for purposes ofremoving a sample of brain tissue. The brain tissue may be subsequentlyassessed by a pathologist to determine if it is cancerous, for example.Brain biopsy procedures may be conducted with or without a stereotacticframe. Both types of procedures may be performed using image-guidance.Frameless biopsies, in particular, may be conducted using the navigationsystem 205.

In some examples, the tracking system 313 may be any suitable trackingsystem. In some examples, the tracking system 313 may be any suitabletracking system which may or may not use camera-based trackingtechniques. For example, a tracking system 313 that does not use thetracking camera, such as a radiofrequency tracking system, may be usedwith the navigation system 205.

In FIG. 4, a block diagram is shown illustrating a control andprocessing system 400 that may be used in the medical navigation system205 shown in FIG. 3 (e.g., as part of the equipment tower). As shown inFIG. 4, in an example, the control and processing system 400 may includeone or more processors 402, a memory 404, a system bus 406, one or moreinput/output interfaces 408, a communications interface 410, and astorage device 412. The control and processing system 400 may beinterfaced with other external devices, such as a tracking system 313,data storage 442, and external user input and output devices 444, whichmay include, for example, one or more of a display, keyboard, mouse,sensors attached to medical equipment, foot pedal, and microphone andspeaker. The data storage 442 may be any suitable data storage device,such as a local or remote computing device (e.g. a computer, hard drive,digital media device, or server) having a database stored thereon. Inthe example shown in FIG. 4, the data storage device 442 includesidentification data 450 for identifying one or more medical instruments460 (e.g., a tracked tool, such as a pointing tool 320) andconfiguration data 452 that associates customized configurationparameters with one or more of the medical instrument(s) 460. The datastorage device 442 may also include preoperative image data 454 and/ormedical procedure planning data 456. Although the data storage device442 is shown as a single device in FIG. 4, it will be understood that inother embodiments, the data storage device 442 may be provided asmultiple storage devices.

The medical instruments 460 may be identifiable by the control andprocessing unit 400. The medical instruments 460 may be connected to andcontrolled by the control and processing unit 400, or the medicalinstruments 460 may be operated or otherwise employed independent of thecontrol and processing unit 400. The tracking system 313 may be employedto track one or more medical instruments 460 and spatially register theone or more tracked medical instruments to an intraoperative referenceframe. For example, the medical instruments 460 may include trackingmarkers 312 as described above with reference to FIG. 3.

The control and processing unit 400 may also interface with a number ofconfigurable devices, and may intraoperatively reconfigure one or moreof such devices based on configuration parameters obtained from theconfiguration data 452. Examples of devices 431, as shown in FIG. 4,include one or more external imaging devices 422, one or moreillumination devices 424, a positioning system 302 (e.g., a roboticarm), an imaging device 412, one or more projection devices 428, one ormore displays 311, and a scanner 420, which in an example may be a 3Dscanner.

Exemplary aspects of the disclosure can be implemented via theprocessor(s) 402 and/or memory 404. For example, the functionalitiesdescribed herein can be partially implemented via hardware logic in theprocessor 402 and partially using the instructions stored in the memory404, as one or more processing modules or engines 470. Exampleprocessing modules include, but are not limited to, a user interfaceengine 472, a tracking module 474, a motor controller 476, an imageprocessing engine 478, an image registration engine 480, a procedureplanning engine 482, a navigation engine 484, and a context analysismodule 486. While the example processing modules are shown separately inFIG. 4, in some examples the processing modules 470 may be stored in thememory 404 and the processing modules 470 may be collectively referredto as processing modules 470. In some examples, two or more modules 470may be used together to perform a function. Although depicted asseparate modules 470, the modules 470 may be embodied as a unified setof computer-readable instructions (e.g., stored in the memory 404)rather than distinct sets of instructions.

It is to be understood that the system is not intended to be limited tothe components shown in FIG. 4. One or more components of the controland processing system 400 may be provided as an external component ordevice. In one example, the navigation module 484 may be provided as anexternal navigation system that is integrated with the control andprocessing system 400.

Some embodiments may be implemented using the processor 402 withoutadditional instructions stored in memory 404. Some embodiments may beimplemented using the instructions stored in memory 404 for execution byone or more general purpose microprocessors. Thus, the disclosure is notlimited to a specific configuration of hardware and/or software.

In some examples, the navigation system 205, which may include thecontrol and processing unit 400, may provide tools to the surgeon thatmay help to improve the performance of the medical procedure and/orpost-operative outcomes. In addition to removal of brain tumours andintracranial hemorrhages (ICH), the navigation system 205 can also beapplied to a brain biopsy, a functional/deep-brain stimulation, acatheter/shunt placement procedure, open craniotomies,endonasal/skull-based/ENT, spine procedures, and other parts of the bodysuch as breast biopsies, liver biopsies, etc. While several exampleshave been provided, examples of the present disclosure may be applied toany suitable medical procedure.

When performing a medical procedure using a medical navigation system205, the medical navigation system 205 typically acquires and maintainsa reference of the location of the tools in use as well as the patientin 3D space. In other words, during a navigated medical procedure, theretypically is a tracked reference frame that is fixed relative to thepatient. For example, during the registration phase of a navigatedneurosurgery, a transformation is calculated that maps the frame ofreference of preoperative magnetic resonance (MR) or computed tomography(CT) imagery to the physical space of the surgery, specifically thepatient's head. This may be accomplished by the navigation system 205tracking locations of fiducial markers fixed to the patient's head,relative to the static patient reference frame. The patient referenceframe is typically rigidly attached to the head fixation device, such asa Mayfield clamp. Registration is typically performed before the sterilefield has been established.

FIG. 5 illustrates a simplified example of how two coordinate spaces maybe co-registered by performing a transformation mapping, based on acommon reference coordinate. In the example shown, a common referencecoordinate 500 has a defined position and orientation in first andsecond coordinate spaces 510, 520. In the context of a medicalprocedure, the common reference coordinate 500 may be a fiducial markeror anatomical reference. Although FIG. 5 illustrates co-registration of2D coordinate spaces, for simplicity, co-registration may be performedfor 3D coordinate spaces, including a depth dimension.

The position and orientation of the common reference coordinate 500 isused to correlate the position of any point in the first coordinatespace 510 to the second coordinate space 520, and vice versa. Thecorrelation is determined by equating the locations of the commonreference coordinate 500 in both spaces 510, 520 and solving for atransformation variable for each degree of freedom defined in the twocoordinate spaces 510, 520. These transformation variables may then beused to transform a coordinate element of a position in the firstcoordinate space 510 to an equivalent coordinate element of a positionin the second coordinate space 520, and vice versa.

In FIG. 5, the common reference coordinate 500 has a coordinate position(x1, y1) determined in the first coordinate space 510 and a coordinateposition (x2, y2) in the second coordinate space 520. In the exampleshown, (x1, y1)=(55, 55) and (x2, y2)=(−45, −25).

Utilizing transformation equations, any point in the first coordinatespace 510 may be related to the second coordinate space 520 viatranslation variables (xT, yT), as shown below:

x1=x2+xT

y1=y2+yT

Using the coordinate positions of the common reference coordinate 500,the transformation variables may be solved as follows:

55=−45+yT

100=yT

55=−25+xT

80=xT

The transformation variables may then be used to transform anycoordinate point in the first coordinate space 510 to the secondcoordinate space 520, and vice versa, thereby co-registering thecoordinate spaces 510, 520. For transformation between 3D coordinatespaces, similar calculations may be performed for position (x, y,z-coordinates) as well as for orientation (pitch, yaw, roll). Ingeneral, a transformation mapping may be performed to register two ormore coordinate spaces with each other. Where there are more than twocoordinate spaces to be co-registered, the transformation mapping mayinclude multiple mapping steps.

In some examples, using a handheld 3D scanner 420, a full or nearly fullarray scan of a surface of interest can be achieved intraoperatively.This may provide an order of magnitude greater point information thanthe surface tracking methods used in conventional approaches. Theintraoperative image data obtained by the 3D scanner 420 may be providedas a 3D point cloud, in an intraoperative image coordinate space. Thispoint cloud may be mapped to a surface in preoperative image data (e.g.,MR or CT volumetric scan data), using a reference marker that isimageable by both preoperative and intraoperative imaging systems. Thetracking system 313 may have no reference to the 3D point cloud data.Therefore, a transformation mapping between the tracking coordinatespace and the intraoperative image coordinate space may be used so thattracking data can also be registered to the preoperative andintraoperative image data.

In the context of the navigation system 205, the co-registration processdescribed above may be used to co-register a tracking coordinate space(which defines the coordinates used by tracking information produced bythe tracking system); a medical image coordinate space (which definesthe coordinates used by medical image data produced by pre-operative orintra-operative imaging, such as MRI or CT); and a camera coordinatespace (which defines the coordinates used by captured image dataproduced by an optical camera). For example, a first transformationmapping may be performed to map two of the three coordinate spaces toeach other (e.g., mapping tracking coordinate space and medical imagecoordinate space to each other), then a second mapping may be performedto map the remaining coordinate space to the first mapping (e.g.,mapping the camera coordinate space to the previous mapping). Thus, acommon coordinate space is obtained in which a first object havingcoordinates defined in one space can be readily related to a secondobject having coordinates defined in another space. In some examples,the common coordinate space may also be referred to as a unifiedcoordinate space.

Methods and systems disclosed herein may provide spatially-accurate andspatially-persistent visual information on a display. This may beenabled by the combined use of the tracked medical instrument (and othertargets), tracked camera and image processing by the navigation system.Tracking of targets enables spatial accuracy, while tracking of thecamera enables spatial persistence. In the present disclosure, the termspatial accuracy may be used to refer to the ability to accurately andprecisely determine the position (e.g., x,y,z-coordinates) andorientation (e.g., φ,θ,ψ angles) of a tracked tool in a certaincoordinate space. The position of an object may generally refer to thecoordinate position of a reference point on the object, such as a distaltip of a pointing tool. The orientation of an object may generally referto the angular orientation of a central axis on the object, such as thecentral longitudinal axis of a pointing tool. The term spatialpersistence may be used to refer to the ability to store and maintainspatial accuracy of a tracked tool in a certain coordinate space even asthe field of view of the camera changes. For example, where a visualindication of a tracked tool is superimposed on an image captured by thecamera, when the field-of-view (FOV) changes (e.g., camera changesposition), the visual indication is updated to reflect the position andorientation of the tracked tool in the new FOV, while maintainingspatial accuracy. That is, information and feedback about the trackedtool is not lost when the FOV changes.

FIG. 6A is a flowchart illustrating an example method 600 for providingfeedback during a medical procedure, for example using the navigationsystem 205 described above. The example method 600 may be implementedduring a neurosurgical procedure, for example as shown in FIG. 7A. Anexample implementation of the method 600 will be described below withreference to FIG. 7A. Other example implementations will also beprovided further below.

The method 600 may take place in the context of an image-guided medicalprocedure. A tracking system 313 (which may be part of the navigationsystem 205) may track a tracked tool, such as a pointing tool havingtracking markers 312, and provide tracking information about the 3Dposition and orientation of the tracked tool during the procedure. Anoptical camera (such as the tracking camera which may be part of thenavigation system 205) may capture an image of the medical procedure.The camera may typically be positioned and oriented to capture a FOV ofthe site, and may be moved to a different position and orientationand/or adjusted to have a different zoom, in order to capture adifferent FOV of the site. A display (such as the display 311 of thenavigation system 205) may be used to display the captured image, andalso to display other navigation information. The method 600 may becarried out by a processor (e.g., in a control and processing system ofthe navigation system 205) coupled to receive the tracking informationfrom the tracking system 313, to receive image data from the camera andto output data to be displayed on the display.

The position and orientation of the camera may be tracked, for exampleby placing tracking markers on the camera and using the tracking system.The tracked position of the camera may be determined relative to thetracking coordinate space and mapped to the common coordinate space. Insome examples, the position and orientation of the camera may bedetermined based on the position of a positioning system (e.g., arobotic arm) where the camera is supported in a known position andorientation relative to the positioning system. For example, thepositioning system may be tracked using tracking markers placed on thepositioning system. In another example, the positioning system mayinclude position sensors which provide information about the position ofthe positioning system. Regardless of how the position and orientationof the camera is determined, this information enables the image capturedby the camera to be mapped to a common coordinate space. In someexamples, calibration of the camera may be performed (e.g., as part ofthe method 600 or prior to the method 600) to map pixel positions of thecaptured image to 3D coordinates in the real world. Any suitable methodmay be used for such calibration.

At 602, the 3D position and orientation of the tracked tool isdetermined. The tracking information from the tracking system is used todetermine the position and orientation of the tracked tool relative tothe site of the procedure. The 3D position and orientation of thetracked tool may be repeatedly determined in real-time by the trackingsystem, so that the tracking information provides real-time informationabout the tracked tool. In the example shown in FIG. 7A, the trackedtool is a pointing tool having tracking markers (e.g., reflectivespheres) detectable by a tracking system. In particular, the trackedpoint may be the distal tip of the pointing tool and the orientation ofthe pointing tool may be defined by the orientation of the centrallongitudinal axis of the pointing tool. Any object detectable by thetracking system may be the tracked tool, including, for example, anyother medical tool such as a suction tool. The tracked point andorientation of the tracked tool may be defined depending on the trackedtool. For example, where the tracked tool has a bent shape (e.g., anL-shaped object), the orientation of the tracked tool may be defined bythe longitudinal axis of the most distal portion of the object.

At 604, the 3D position and orientation of the tracked tool is mapped tothe common coordinate space. This may be performed by transforming thetracking information from the coordinate space of the tracking system tothe common coordinate space. As described previously reference points onthe surgical site may also be mapped to the common coordinate space. Aswell, the FOV of the camera is also mapped to the common coordinatespace (e.g., by tracking the position and orientation of the opticalcamera, using the tracking system and mapping the resulting informationto the common coordinate space). Hence, the real-time 3D position andorientation of the tracked tool can be related to the surgical site andalso to the FOV of the camera.

At 606, optionally, a selection of a 3D point is received. The 3D pointmay be selected by positioning the tracked tool at a desired locationand activating an input mechanism to select the point. For example, thedistal tip of the pointing tool may be placed at a desired location andan input mechanism (e.g., a button on the pointing tool or a foot pedalcoupled to the navigation system) may be activated to indicate selectionof the position of the distal tip as the selected 3D point.

In some examples, the 3D point may be selected by interacting with thedisplay of the captured image. For example, the surgeon may move acursor over the displayed image and click on a point on the image, ormay touch a touch-sensitive display at a point on the image. FIG. 6Bshows an example in which a cursor 652 is manoeuvred in the displayedimage 650. In the example shown, a menu 656 provides selectable optionsfor selecting 3D points or reference lines. The surgeon may interactwith the image 650 (e.g., clicking when the cursor 652 is at the desiredlocation) to select a point 654 on the image 650. Because the displayedimage 650 is a 2D image (i.e., having only x,y-coordinates), it may beassumed that the depth (i.e., z-coordinate) of the selected point 654corresponds to the depth of the tissue displayed at the selected point654. The depth of the tissue at any point may be determined using anysuitable technique including, for example by obtaining a 3D scan of thetissue (e.g., using a 3D scanner 420) or by performing image analysis(e.g., by analyzing the focus depth at which the tissue is in focus). Insome examples, the depth of the tissue may be determined frompreoperative image data, such as MR or CT image data. In such cases, thedepth of the selected point may correspond to deeper structures belowthe tissue surface, for example where the preoperative image datacaptures data about a structure (e.g., a tumor) below the tissuesurface. Using the common coordinate space, the selected point 654 onthe displayed image can be transformed to a 3D point in the trackingcoordinate space. Thus, a point 654 that is selected by interacting withthe displayed image 650 may be processed the same way as a point that isselected in 3D space by selection using a tracked pointing tool. Forsimplicity, the examples discussed below will refer to selection using atracked tool in the tracking coordinate space. However, it should beunderstood that such examples may be similarly carried out using pointsselected by interacting with the displayed image, or a combination ofselection methods.

In some examples, a single interaction may be used to select multiple 3Dpoints or regions. For example, a single selection may be made to selectall portions of the FOV corresponding to a characteristic of a selectedpoint, such as the colour or depth indicated by the distal tip of thepointing tool.

The selected 3D point is stored in memory. In some examples, in additionto the 3D position of the 3D point, the orientation of the tracked toolis also stored. For example, the 3D position of the distal tip may bestored in association with the 3D orientation of the longitudinal axisof the pointing tool. The selected 3D point may thus be stored inassociated with the selected 3D orientation.

In some examples, there may not be a selection of a 3D point and 606 maybe omitted. In such cases, the following steps of the method 600 may beperformed for the tracked real-time 3D position and/or orientation ofthe tracked tool. In some examples, even when a 3D point has beenselected at 606, the method 600 may additionally be performed for thereal-time 3D position and/or orientation of the tracked tool, forexample as illustrated by examples discussed further below.

At 608, navigational information associated with the 3D position (andoptionally orientation) of the tracked tool (and optionally the selected3D point) is determined. In some examples, other sets of data (e.g.,previously selected 3D points) may be used for determining thenavigational information. In some examples, navigational information maybe simply the 3D location and optionally orientation of the tracked tool(and optionally the selected 3D point) relative to the surgical site andthe FOV of the camera.

At 610, a representation of the navigational information is displayed.This may involve the processor generating a virtual representation ofthe navigational information and outputting data to superimpose thevirtual representation on the optical image captured by the camera. Thevirtual representation may be generated using the common coordinatespace, so that the representation is superimposed on the optical imagein a location of the image appropriate to the navigational information.

If no 3D point was selected (606 was omitted), the displayednavigational information may be navigational information related to thereal-time 3D position (and optionally orientation) of the tracked tool.Navigational information related to the real-time 3D position and/ororientation may be a representation of the 3D position and/ororientation relative to the surgical site. Navigational informationrelated to the real-time tracked position may be referred to as dynamicnavigational information because it is dependent on real-time positionof the tracked tool.

If a 3D point was selected at 606, the displayed navigationalinformation may additionally or alternatively include navigationalinformation calculated based on the selected 3D point. Navigationalinformation related to a selected point may be referred to as staticnavigational information because it is dependent on a selected pointthat is not time-dependent. The displayed navigational information mayinclude dynamic navigational information, static navigationalinformation, and combinations thereof.

For example, the displayed representation of the navigationalinformation may include a crosshair representing the projection of thedistal tip of the tracked tool onto the surface of the surgical site. Inanother example, the displayed representation may include a linerepresenting the longitudinal axis of the tracked tool.

In the example of FIG. 7A, the navigational information that isdetermined is the distance between two selected 3D points, as well asthe location of the 3D points relative to the surgical site. Thedistance information may be determined by calculating the 3D distancebetween two selected 3D points, such as a currently selected 3D point702 and an immediately previous selected 3D point 704.

In the example of FIG. 7A, the two 3D points 702, 704 are represented bytwo dots superimposed on the captured image 750, corresponding to the 3Dposition of the 3D points 702, 704 relative to the surgical site. Thedistance between the two 3D points 702, 704 is represented by a linebetween the two points 702, 704, and a label 706 indicating thedistance.

In some examples, the distance may be calculated between one 3D point702 and a predefined point (e.g., a predefined surgical target) insteadof a previously selected 3D point. In some examples, the distance may becalculated between the 3D point 702 and a reference depth plane. Areference depth plane may be predefined (e.g., zero depth may bepredefined as the surface of the patient's skin) or may be defined to bethe depth of a previously selected 3D point. The orientation of thereference depth plane may be predefined or defined according to theorientation of the pointing tool, for example.

FIG. 7C illustrates an example, in the context of a spinal procedure, ofhow a reference depth plane 722 may be defined by a previously selected3D point 724, and the navigational information may be the depth of acurrently selected 3D point 726 relative to the reference depth plane722 (e.g., calculated as a perpendicular distance from the referencedepth plane 722).

In some examples, instead of a selected 3D point 702, the distance ordepth may be calculated between a previously selected 3D point 704 (or areference point or plane) and the real-time 3D position of the trackedtool (in this example the distal tip of a pointing tool).

Other navigational information that may be calculated include, forexample, angle measurements between two orientations (e.g., between aselected 3D orientation and a planned trajectory line, between twoselected 3D orientations, or between a selected 3D orientation and areal-time tracked 3D orientation).

Such visuospatial information may be useful for collection of anatomicmeasurements in real-time (e.g., disc space height/depth or relativeangle of vertebral endplates during a spinal procedure), and fordetermining changes in such anatomic measurements during the procedure(e.g., in discectomy and distraction). Calculation and display of thisinformation based on selection by a pointing tool may simplify theprocedure and may provide more accurate information, compared toconventional techniques (e.g., physically placing a rule on the targetarea or using X-ray imaging to confirm desired anatomic corrections).

At 612, the displayed representation is updated when: the 3D positionand orientation of the tracked tool changes (614); and/or when the FOVof the camera changes (616).

614 may be carried out where the navigational information is dynamicnavigational information dependent on the real-time tracking of thetracked tool. For example, updating the displayed dynamic navigationalinformation may include performing 602, 604 and 608 to track the objectand calculate navigational information as the object moves, and thendisplaying the updated navigational information. The updatednavigational information may be an updated representation of the 3Dposition and/or orientation of the tracked tool relative to the surgicalsite. For example, distance or depth relative to a reference point orplane may be calculated and updated in real-time as the tracked toolmoves. Other examples will be discussed further below.

616 may be carried out for both dynamic and static navigationalinformation, to reflect the changed FOV and to maintain spatialpersistence of the representation in the changed FOV. Changes in the FOVof the camera may be determined by the tracking information from thetracking system (e.g., in examples where the camera is tracked by thetracking system), by information from a positioning system thatpositions the camera (e.g., in examples where the camera is supported bya robotic arm) and/or by information from the camera itself (e.g., thecamera may provide information indicating the zoom level of the capturedimage). Because the 3D point, surgical site and the captured image areall mapped to the common coordinate space, the visual representation canbe updated by the processor.

In some examples, updating of the displayed representation may also beperformed at fixed time intervals (e.g., every 100 ms) or in response touser input. Thus, an update (or refresh) of the displayed representationmay occur even where there is no movement of the tracked tool and nochange in FOV of the camera.

In the example of FIG. 7A, when the FOV changes, the representation ofthe 3D points 702, 704 is updated to accurately depict the 3D positionof the 3D points 702, 704 within the new FOV. An example is illustratedin FIGS. 9B-9G (discussed in greater detail below), where selectedpoints shown in FIGS. 9C and 9E are persistent even when the viewpointchanges in FIG. 9G. Where the orientation and/or the zoom level of thecamera changes, the visual representation of the distance between thepoints 702, 704 may change (e.g., visually lengthened when the zoomlevel increases), however because the actual physical distance betweenthe points 702, 704 is unchanged the distance indicated by the label 706is unchanged. An example of this is shown in FIG. 7B. In one image 760a, the image is shown at a first zoom level, including visualrepresentation of 3D points 702, 704 and a label 706 indicating theactual distance between the points 702, 704. When the zoom level isincreased to the second image 760 b, the visual representation of the 3Dpoints 702, 704 and the distance between them is accordingly alsozoomed, however the actual distance indicated by the label 706 isunchanged. The processor may perform calculations to update the visualrepresentation in accordance with the changed FOV, but the processordoes not need to recalculate the 3D position of the points 702, 704 orthe navigational information (in this case, the distance between thepoints 702, 704) because no changes have been made to the physicalposition and orientation of the selected 3D points.

In some examples, the orientation as well as position of the 3D pointmay be determined and stored. The visual representation of the selected3D point may include information indicating the selected 3D orientationof the tracked tool. For example, FIG. 9A shows an image in which theselected 3D point is represented as an arrow superimposed on thecaptured image 950, where the tip of the arrow 902 represents theposition of the selected 3D point, and the body of the arrow correspondsto the selected 3D orientation.

FIGS. 9B-9G illustrate an example of how selected 3D points andassociated 3D orientations are provided as visuospatial information.FIGS. 9B-9G also illustrate an example in which a 3D point may beselected without the use of a tracked tool. In FIGS. 9B-9G, a 3D pointmay be selected as the center of the captured image, and the associated3D orientation may be the normal to the plane of the captured image. InFIGS. 9B and 9C, a first 3D point 902 a is selected (e.g., by activationof an input mechanism such as a foot pedal) when the FOV of the camera(e.g., an topical scope 304) is at a first position and orientation(indicated by dotted line), corresponding to the captured image 960 ashown in FIG. 9C. In FIGS. 9D and 9E, a second 3D point 904 a isselected in a similar way when the FOV of the camera is at a secondposition and orientation (indicated by dotted line), corresponding tothe captured image 960 b shown in FIG. 9E. The 3D positions andassociated orientations are stored. In FIGS. 9F and 9G, the camera againchanges FOV to capture a side view of the surgical site, correspondingto the captured image 960 c shown in FIG. 9G. In FIG. 9G, the first andsecond 3D points 902 a, 904 a are superimposed on the captured image.Further, the 3D orientation associated with the stored points arerepresented as axes 902 b, 904 b.

FIG. 8 illustrates another example of visuospatial information that maybe provided as feedback during a medical procedure. In this example, aplurality of 3D points may be selected in order to form a boundary ofinterest (BOI) 802. The BOI may be defined by connecting the 3D pointsalong the shortest path between points (e.g., creating a closed polygonshape), or may be defined by performing a spline interpolation of thepoints (e.g., to obtain a smoother BOI), for example. The BOI mayfurther define a region of interest (ROI), as discussed further below.The plurality of 3D points may be selected by the surgeon positioningthe pointing tool at different locations and activating the inputmechanism at each location. The plurality of 3D points may also beselected by the surgeon maintaining activation of the input mechanism(e.g., keeping a button or foot pedal depressed) while moving thepointing tool to “draw” the boundary in 3D space. The 3D position of thedistal tip of the pointing tool may be sampled at regular intervalswhile the input mechanism is activated, to obtain a set of 3D pointsthat is used to define the BOI. To assist in selection of points for theBOI, the display may be updated with a visual representation of the 3Dpoints as they are selected.

A visual representation of the BOI 802 may be superimposed on thecaptured image 850, as discussed above. Navigational informationassociated with the BOI 802 may include imaging data obtained prior toor during the procedure. For example, the navigational information mayinclude pre-operative and/or intra-operative imaging such as ultrasound,or 3D imaging of blood vessels or nervous structures. Through the use ofa common coordinate space (e.g., using transformation mapping asdiscussed above), the portion of pre-surgical imaging data thatcorresponds to the ROI defined by the BOI 802 may be identified andextracted. The imaging data 1004 that is provided as visuospatialinformation overlaid on the captured image 1050 may thus be limited tothe ROI defined by the BOI 802, as shown in FIG. 10. This may reduce thecognitive load on the surgeon by presenting only navigationalinformation that is relevant to the ROI.

FIG. 12 illustrates another example in which the visual representationof the ROI defined by the BOI 802 is modified. Here, modification to theimage characteristics of the ROI (e.g., brightness, hue and/orsaturation) is made to the captured image 1250. For example, the ROI maybe modified to remove redness, so that the site can be better viewedwithout being obscured by blood. By limiting such modification to theROI, rather than the entire captured image 1250, the processing load onthe processor may be decreased and performance may be improved. In someexamples, the visual modification may include an automatic change to theposition, orientation and/or zoom level of the camera so that the ROI iskept within the FOV.

In some examples, feedback may be provided to indicate whether a trackedtool (e.g., surgical tool) is within, on or outside of the BOI 802. Forexample, in a training context, a trainer may select a BOI 802 to definethe region within which a trainee should operate. If the trainee movesthe tracked surgical tool out of the BOI 802, feedback (e.g., an audiocue) may be provided to warn the trainee to stay within the BOI 802.

In some examples, the visuospatial feedback may be provided as anoverlay of the real-time optical image superimposed on imaging data, forexample as shown in FIG. 11. In this example, the portion of imagingdata (in this case, an intra-operative 3D scan 1150 of the surface ofthe patient's head) corresponding to a selected ROI (or corresponding tothe vicinity of a selected 3D point) is identified, using the commoncoordinate space. The identified portion of imaging data is thenoverlaid with a portion of the real-time optical image 1104corresponding to the selected ROI (or vicinity of the selected 3Dpoint). It should be noted that the intra-operative 3D scan 1150 isoriented to match the orientation of the optical image 1104, based onmapping to the common coordinate space.

FIG. 13 illustrates an example in which the feedback provided is in theform of reference lines 1302 superimposed on the captured image 1350. Areference line 1302 may be defined by connecting between two selected 3Dpoints, in a manner similar to how a BOI is defined. A reference line1302 may additionally or alternatively be defined by a selected 3Dorientation (e.g., defined by the longitudinal axis of the trackedtool). The reference lines 1302 may be used, for example, to alignscrews in lumbar fusion surgery. It should be noted that the referenceline 1302 may also be dynamic, that is the reference line 1302 may bedefined by the real-time orientation of the tracked tool. Furthernavigational information that may be represented may be an anglemeasurement 1304 between two reference lines.

FIG. 14 illustrates an example in which the navigational information isprovided as a visual representation (in this case a cube 1404) of theorientation of the captured image 1450 relative to a referenceorientation (e.g., the patient's anatomical orientation). The cube 1404may show symbols indicating orientation directions. For example, thecube 1404 may show “H” for head, “R” for right and “A” for anterior. Asthe orientation of the FOV changes, the cube 1404 also changes torepresent the corresponding reference orientation.

In some examples, navigational information may be based on informationextracted from planning information. Planning information may includeinformation defining a planned trajectory and/or identification of oneor more planned targets or reference points. Such pre-surgical planningmay be carried out using pre-surgical imaging data, and defined in theimaging data coordinate space. Using transformation to the commoncoordinate space, the planning information may be provided asvisuospatial information overlaid on the captured image. Points orregions of interest may also be selected, pre-operatively orintra-operatively, in the imaging data coordinate space (e.g., byinteracting with a displayed MRI image) and similarly correlated to thecaptured image.

An example of this is shown in FIG. 15. Here, points 1502 identified inthe imaging data coordinate space are superimposed on the captured image1550. The identified points 1502 may be labelled according to labelsdefined in planning information. In some examples, the identified points1502 may be labelled according to user input (e.g., the surgeon mayselect or enter a label for a point when a 3D point is selected). Inthis example, the points 1502 are labelled as “TP” for transverseprocess and “SP” for spinous process. In another example, a plannedtrajectory may be displayed as an arrow or path overlaid on the capturedimage 1550. This visual feedback may be combined with other visualmodifications, such as changes in colour and/or size to indicate whethera tracked tool is correctly positioned or aligned with the plannedtrajectory/target. The visual feedback may also be combined with otherfeedback modalities, such as audio cues to indicate if the tracked toolis properly positioned or aligned. By providing planning information insitu as visuospatial feedback, performance error may be reduced.

FIG. 16 illustrates an example of a user interface 1604 that may bepresented, to enable the surgeon to interact with a navigation systemusing a tracked tool 320 (in this example, a pointing tool). The userinterface 1604 may be presented as a radial menu overlaid on thecaptured image 1650 and centered about the tracked tool. Icons in theradial menu may be selected to control various aspects of the navigationsystem, for example to change a zoom level of the optical camera, tochange the type of visuospatial information presented and/or to causedisplay of other information on another display. By changing theorientation of the tracked tool and without moving the distal point ofthe tool, the surgeon may select a particular icon in the user interface1604. This may enable the surgeon to more easily provide input to thenavigation system, without having to change to a different display orotherwise remove attention from the surgical site.

It should be understood that the various examples of visuospatialinformation described above may be provided in combination. In someexamples, it may be possible to switch between different displays ofvisuospatial information. It may be possible to select whether or not todisplay certain selected 3D point(s), BOI(s) and/or reference line(s),and 3D point(s), BOI(s) and/or reference line(s) may be selectivelydeleted or removed from memory.

In some examples, selection of a 3D point may be performed with a toolother than a pointing tool. For example, any tracked surgical tool maybe used to select a 3D point. In another example, when a selection inputis made and there is no tracked tool within the FOV of the camera, thecenter of the captured image may be selected by default, and theselected orientation may be the normal to the plane of the capturedimage by default. As well, selection of a 3D point may be performedthrough interaction with a displayed optical image, or throughinteraction with other imaging data.

Further, selection of a 3D point may not be required for navigationalinformation to be calculated and displayed. The navigational informationmay be calculated and displayed based only on the real-time trackedposition and orientation of the tracked tool.

In some examples, navigational information associated with the real-timetracked position and orientation, selected 3D point(s), defined BOI(s)and/or reference line(s) may be presented using other feedbackmodalities, including tactile feedback and audio feedback, for example.The selected 3D point(s), defined BOI(s) and/or reference line(s) pointmay also be represented in other visual feedback modalities. Forexample, the selected 3D point(s), defined BOI(s) and/or referenceline(s) may also be displayed as a visual overlay on a 3D scan or in anMRI image. Similarly, 3D point(s), defined BOI(s) and/or referenceline(s) that are selected in other modalities (e.g., through interactingwith an image of a 3D scan or an MRI image) may also be displayed as avisual overlay in the captured optical image. In this way, the presentdisclosure provides spatial persistence not only within a singlefeedback modality, but also spatial persistence across multiple imagingmodalities. FIG. 17 shows an example in which the visual representationof the navigational information is persistent across different imagemodalities. In this example, visual representation of selected 3D points1702 is persistent between a preoperative image 1710 and the real-timeoptically captured image 1720. Notably, the locations of the selected 3Dpoints 1702 are spatially persistent across the different images 1710,1720.

In various examples disclosed herein, the present disclosure providesnavigational information to the surgeon in the context of the displayedoptical image. The surgeon is not required to switch tools (e.g., use aphysical rule to measure distances), refer to another interface (e.g.,refer to a separate screen showing navigational information) orotherwise interrupt the procedure in order to access navigationalinformation. Although examples above describe using a pointing tool asthe tracked tool, any tool held in the surgeon's hand may serve as thetracked tool. For example, the distal tip of any tool (e.g., where thedistal tip position has been determined relative to the tracked tool,via calibration) may be used similar to the distal tip of the pointingtool. Thus, the surgeon is able to access more information while keepingthe same tool held in the hand.

Further, by providing the navigational information displayed on theoptical image, other personnel in the operating room may be able to viewthe navigational information, for example for training purposes. Theoptical image with superimposed navigational information may also bestored for future use (e.g., for quality assurance purposes).

It should be understood that the captured optical images in the variousexamples described above may be real-time video images.

Although the above discussion refers to the surgeon as being the userwho controls and uses the examples of the present disclosure, it shouldbe understood that the present disclosure is not limited to any specificuser. In some examples, there may be a plurality of users involved.

While some embodiments or aspects of the present disclosure may beimplemented in fully functioning computers and computer systems, otherembodiments or aspects may be capable of being distributed as acomputing product in a variety of forms and may be capable of beingapplied regardless of the particular type of machine or computerreadable media used to actually effect the distribution.

At least some aspects disclosed may be embodied, at least in part, insoftware. That is, some disclosed techniques and methods may be carriedout in a computer system or other data processing system in response toits processor, such as a microprocessor, executing sequences ofinstructions contained in a memory, such as read-only memory (ROM),volatile random access memory (RAM), non-volatile memory, cache or aremote storage device.

A computer readable storage medium may be used to store software anddata which when executed by a data processing system causes the systemto perform various methods or techniques of the present disclosure. Theexecutable software and data may be stored in various places includingfor example ROM, volatile RAM, non-volatile memory and/or cache.Portions of this software and/or data may be stored in any one of thesestorage devices.

Examples of computer-readable storage media may include, but are notlimited to, recordable and non-recordable type media such as volatileand non-volatile memory devices, ROM, RAM, flash memory devices, floppyand other removable disks, magnetic disk storage media, optical storagemedia (e.g., compact discs (CDs), digital versatile disks (DVDs), etc.),among others. The instructions can be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, and the like. The storage medium may be the internet cloud, ora computer readable storage medium such as a disc.

Furthermore, at least some of the methods described herein may becapable of being distributed in a computer program product comprising acomputer readable medium that bears computer usable instructions forexecution by one or more processors, to perform aspects of the methodsdescribed. The medium may be provided in various forms such as, but notlimited to, one or more diskettes, compact disks, tapes, chips, USBkeys, external hard drives, wire-line transmissions, satellitetransmissions, internet transmissions or downloads, magnetic andelectronic storage media, digital and analog signals, and the like. Thecomputer useable instructions may also be in various forms, includingcompiled and non-compiled code.

At least some of the elements of the systems described herein may beimplemented by software, or a combination of software and hardware.Elements of the system that are implemented via software may be writtenin a high-level procedural language such as object oriented programmingor a scripting language. Accordingly, the program code may be written inC, C++, J++, or any other suitable programming language and may comprisemodules or classes, as is known to those skilled in object orientedprogramming. At least some of the elements of the system that areimplemented via software may be written in assembly language, machinelanguage or firmware as needed. In either case, the program code can bestored on storage media or on a computer readable medium that isreadable by a general or special purpose programmable computing devicehaving a processor, an operating system and the associated hardware andsoftware that is necessary to implement the functionality of at leastone of the embodiments described herein. The program code, when read bythe computing device, configures the computing device to operate in anew, specific and predefined manner in order to perform at least one ofthe methods described herein.

While the teachings described herein are in conjunction with variousembodiments for illustrative purposes, it is not intended that theteachings be limited to such embodiments. On the contrary, the teachingsdescribed and illustrated herein encompass various alternatives,modifications, and equivalents, without departing from the describedembodiments, the general scope of which is defined in the appendedclaims. Except to the extent necessary or inherent in the processesthemselves, no particular order to steps or stages of methods orprocesses described in this disclosure is intended or implied. In manycases the order of process steps may be varied without changing thepurpose, effect, or import of the methods described.

1. A system for providing feedback during a medical procedure, thesystem comprising: a tracking system configured to obtain trackinginformation about three-dimensional (3D) position and orientation of atracked tool during the medical procedure; a camera for capturing anoptical image of a field-of-view (FOV) of a site and the tracked toolduring the medical procedure; a display for displaying the optical imageof the FOV; and an input mechanism for indicating selection of one ormore virtual 3D points in the optical image of the FOV, the inputmechanism being part of the tracked tool; a processor coupled to receiveinput data from the tracking system, the input mechanism, and thecamera, and coupled to transmit output data for display on the display,the processor being configured to: determine the 3D position andorientation of the tracked tool, relative to the site, based on thetracking information; receive selection of one or more 3D points fromthe input mechanism; map the 3D position and orientation of the trackedtool and the one or more selected 3D points to the FOV; determinenavigational information comprising a measurement of distance or anglebetween the 3D position and orientation of the tracked tool to areference defined by at least one of the one or more selected 3D pointsin the site; cause the display to display a virtual representation ofthe navigational information overlaid on the FOV of the site and thetracked tool, the navigational information being displayed as anindication of the measurement overlaid on the FOV.
 2. The system ofclaim 1, wherein the processor is further configured to: update thedisplayed virtual representation by: when the 3D position andorientation of the tracked tool changes, updating the displayed virtualrepresentation of the one or more selected 3D points in accordance withthe changed 3D position and orientation of the tracked tool; or when theFOV changes, updating the displayed virtual representation of the one ormore selected 3D points to follow the changed FOV.
 3. The system ofclaim 2, wherein the tracking system is further configured to: trackposition and orientation of the camera or a support of the camera; andtrack position and orientation of the site; wherein the processor isfurther configured to: map between the 3D position and orientation ofthe tracked tool, the site and the FOV, using information from thetracking system; and cause the display to update the displayed virtualrepresentation to follow the changed FOV, based on mapping between the3D position and orientation of the tracked tool, the site and the FOV,using information from the tracking system.
 4. The system of claim 3,wherein the processor is further configured to: determine an orientationof the camera relative to a reference orientation, using informationfrom the tracking system; and cause the display to display informationoverlaid on the FOV indicating the orientation of the FOV relative tothe reference orientation.
 5. The system of claim 1, wherein theprocessor is configured to: receive selection of multiple 3D pointsindicated by the input mechanism; determine a region of interest (ROI)based on the selected multiple 3D points; determine navigationalinformation associated with the ROI; and cause the display to display avirtual representation of the ROI overlaid on the FOV, the virtualrepresentation including the navigational information.
 6. The system ofclaim 5, wherein the navigational information comprises imaging data ofthe ROI, wherein the overlay on the FOV includes the imaging data onlyfor the ROI.
 7. The system of claim 5, wherein the processor is furtherconfigured to modify visual display of the optical image of the FOVwithin the ROI.
 8. The system of claim 1, wherein the processor isfurther configured to: determine navigational information by determiningorientation of a longitudinal axis of the tracked tool; and cause thedisplay to display a virtual representation of the navigationalinformation as a reference line overlaid on the FOV.
 9. The system ofclaim 1, wherein the measurement is a measurement of the distancebetween the 3D position of the tracked tool and the at least one of theone or more selected 3D points.
 10. The system of claim 1, wherein theat least one of the one or more selected 3D points defines part of areference line, and the measurement is a measurement of an angle betweenthe 3D orientation of the tracked tool and the reference line.
 11. Thesystem of claim 1, wherein the at least one of the one or more selected3D points defines part of a reference depth plane, and the measurementis a measurement of a depth difference between the 3D position of thetracked tool and the reference depth plane.
 12. The system of claim 1,wherein the processor is further configured to: receive planninginformation defining a planned trajectory or planned target; wherein thenavigational information is extracted from the planning information anddisplayed in association with the 3D position and orientation of thetracked tool overlaid on the FOV.
 13. The system of claim 1, wherein theprocessor is further configured to: cause the display to display avirtual representation of the 3D position and orientation of the trackedtool overlaid on a display of imaging data.
 14. The system of claim 13,wherein the processor is further configured to: cause the display todisplay a portion of the FOV, corresponding to the 3D position andorientation of the tracked tool, overlaid on the imaging data.
 15. Thesystem of claim 1, wherein the system comprises an additional inputmechanism for selecting an additional virtual 3D point via userinteraction with the displayed FOV, and wherein the processor is furtherconfigured to: receive selection of the additional virtual 3D point viauser interaction with the displayed FOV.
 16. The system of claim 1,wherein the one or more selected 3D points are stationary relative tothe site.
 17. A method for providing feedback during a medicalprocedure, the method comprising: determining a 3D position andorientation of a tracked tool, relative to a site of the medicalprocedure, based on tracking information received from a tracking systemthat is tracking the tracked tool; receiving selection of one or more 3Dpoints from an input mechanism, the input mechanism being part of thetracked tool; mapping the 3D position and orientation of the trackedtool and the one or more selected 3D points to a field-of-view (FOV) ofa camera that is capturing an optical image of the site and the trackedtool; determining navigational information comprising a measurement ofdistance or angle between the 3D position and orientation of the trackedtool to a reference defined by at least one of the one or more selected3D points in the site; causing a display to display a virtualrepresentation of the navigational information overlaid on the FOV ofthe site and the tracked tool, the navigational information beingdisplayed as an indication of the measurement overlaid on the FOV. 18.The method of claim 17, further comprising: updating the displayedvirtual representation by: when the 3D position and orientation of thetracked tool changes, updating the displayed virtual representation ofthe one or more selected 3D points in accordance with the changed 3Dposition and orientation of the tracked tool; or when the FOV changes,updating the displayed virtual representation of the one or moreselected 3D points to follow the changed FOV.
 19. The method of claim18, further comprising: tracking a position and orientation of thecamera or a support of the camera, and tracking a position andorientation of the site with the tracking system; and mapping the 3Dposition and orientation of the tracked tool, the site and the FOV,using information from the tracking system; and updating the displayedvirtual representation on the display to follow the changed FOV, basedon the mapping between the 3D position and orientation of the trackedtool, the site and the FOV, using information from the tracking system.20. The method of claim 19, further comprising: receiving selection ofan additional virtual 3D point via an additional input mechanism;mapping the additional virtual 3D point to the FOV; wherein thereference is further defined by the additional virtual 3D point in thesite.